<!DOCTYPE html>
<html id="docs" lang="en" class="">
	<head>
	<meta charset="utf-8">
<title>Coarse Parallel Processing Using a Work Queue - Kubernetes</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" type="image/png" href="../../../../images/favicon.png">
<link rel="stylesheet" type="text/css" href="../../../../css/base_fonts.css">
<link rel="stylesheet" type="text/css" href="../../../../css/styles.css">
<link rel="stylesheet" type="text/css" href="https://code.jquery.com/ui/1.12.1/themes/smoothness/jquery-ui.css">
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.css">
<link rel="stylesheet" type="text/css" href="../../../../css/callouts.css">
<link rel="stylesheet" type="text/css" href="../../../../css/custom-jekyll/tags.css">




<meta name="description" content="Coarse Parallel Processing Using a Work Queue" />
<meta property="og:description" content="Coarse Parallel Processing Using a Work Queue" />

<meta property="og:url" content="https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/" />
<meta property="og:title" content="Coarse Parallel Processing Using a Work Queue - Kubernetes" />

<script
src="https://code.jquery.com/jquery-3.2.1.min.js"
integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
crossorigin="anonymous"></script>
<script
src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"
integrity="sha256-VazP97ZCwtekAsvgPBSUwPFKdrwD3unUfSGVYrahUqU="
crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.js"></script>
<script src="../../../../js/script.js"></script>
<script src="../../../../js/custom-jekyll/tags.js"></script>


	</head>
	<body>
		<div id="cellophane" onclick="kub.toggleMenu()"></div>

<header>
    <a href="../../../../index.html" class="logo"></a>

    <div class="nav-buttons" data-auto-burger="primary">
        <ul class="global-nav">
            
            
            <li><a href="../../../home.1">Documentation</a></li>
            
            <li><a href="../../../../blog/index.html">Blog</a></li>
            
            <li><a href="../../../../partners/index.html">Partners</a></li>
            
            <li><a href="../../../../community/index.html">Community</a></li>
            
            <li><a href="../../../../case-studies/index.html">Case Studies</a></li>
            
            
             <li>
                <a href="index.html#">
                    English <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="../../../../zh/index.html">中文 Chinese</a></li>
                
                    <li><a href="../../../../ko/index.html">한국어 Korean</a></li>
                
                </ul>
            </li>
         
            <li>
                <a href="index.html#">
                    v1.11 <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="https://kubernetes.io">v1.12</a></li>
                
                    <li><a href="../../../../index.html">v1.11</a></li>
                
                    <li><a href="https://v1-10.docs.kubernetes.io">v1.10</a></li>
                
                    <li><a href="https://v1-9.docs.kubernetes.io">v1.9</a></li>
                
                </ul>
            </li>
        </ul>
        
        <a href="../../../tutorials/kubernetes-basics/index.html" class="button" id="tryKubernetes" data-auto-burger-exclude>Try Kubernetes</a>
        <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button>
    </div>

    <nav id="mainNav">
        <main data-auto-burger="primary">
        <div class="nav-box">
            <h3><a href="../../../tutorials/stateless-application/hello-minikube/index.html">Get Started</a></h3>
            <p>Ready to get your hands dirty? Build a simple Kubernetes cluster that runs "Hello World" for Node.js.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../home.1">Documentation</a></h3>
            <p>Learn how to use Kubernetes with the use of walkthroughs, samples, and reference documentation. You can even <a href="../../../../editdocs/index.html" data-auto-burger-exclude>help contribute to the docs</a>!</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../../community/index.html">Community</a></h3>
            <p>If you need help, you can connect with other Kubernetes users and the Kubernetes authors, attend community events, and watch video presentations from around the web.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../../blog/index.html">Blog</a></h3>
            <p>Read the latest news for Kubernetes and the containers space in general, and get technical how-tos hot off the presses.</p>
        </div>
        </main>
        <main data-auto-burger="primary">
        <div class="left">
            <h5 class="github-invite">Interested in hacking on the core Kubernetes code base?</h5>
            <a href="https://github.com/kubernetes/kubernetes" class="button" data-auto-burger-exclude>View On Github</a>
        </div>

        <div class="right">
            <h5 class="github-invite">Explore the community</h5>
            <div class="social">
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>Twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
        </div>
        <div class="clear" style="clear: both"></div>
        </main>
    </nav>
</header>

		
		
		<section id="hero" class="light-text no-sub">
			











<h1>Tasks</h1>
<h5></h5>






<div id="vendorStrip" class="light-text">
	<ul>
		
		
		<li><a href="../../../home.1">DOCUMENTATION</a></li>
		
		
		<li><a href="../../../setup/index.html">SETUP</a></li>
		
		
		<li><a href="../../../concepts/index.html">CONCEPTS</a></li>
		
		
		<li><a href="../../index.html" class="YAH">TASKS</a></li>
		
		
		<li><a href="../../../tutorials/index.html">TUTORIALS</a></li>
		
		
		<li><a href="../../../reference.1">REFERENCE</a></li>
		
	</ul>
	<div id="searchBox">
		<input type="text" id="search" placeholder="Search" onkeydown="if (event.keyCode==13) window.location.replace('/docs/search/?q=' + this.value)" autofocus="autofocus">
	</div>
</div>

		</section>
		
		
<section id="deprecationWarning">
  <main>
    <div class="content deprecation-warning">
      <h3>
        Documentation for Kubernetes v1.11 is no longer actively maintained. The version you are currently viewing is a static snapshot.
        For up-to-date documentation, see the <a href="https://kubernetes.io/docs/home/">latest</a> version.
      </h3>
    </div>
  </main>
</section>


		<section id="encyclopedia">
			
<div id="docsToc">
     <div class="pi-accordion">
    	
        
        
        
        
        
         
             
                 
             
         
             
                 
             
         
             
                 
             
         
             
                 
             
         
             
                 
                          
                          
                 
             
         
             
         
             
         
             
         
         
        
        <a class="item" data-title="Tasks" href="../../index.html"></a>

	
	
		
		
	<div class="item" data-title="Install Tools">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Install and Set Up kubectl" href="../../kubectl/install/index.html"></a>

		
	
		
		
<a class="item" data-title="Install Minikube" href="../../tools/install-minikube/index.html"></a>

		
	
		
		
<a class="item" data-title="Installing kubeadm" href="../../../setup/independent/install-kubeadm/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Configure Pods and Containers">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Assign Memory Resources to Containers and Pods" href="../../configure-pod-container/assign-cpu-ram-container"></a>

		
	
		
		
<a class="item" data-title="Assign CPU Resources to Containers and Pods" href="../../configure-pod-container/assign-cpu-resource/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Quality of Service for Pods" href="../../configure-pod-container/quality-service-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Assign Extended Resources to a Container" href="../../configure-pod-container/extended-resource/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a Volume for Storage" href="../../configure-pod-container/configure-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a PersistentVolume for Storage" href="../../configure-pod-container/configure-persistent-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a Projected Volume for Storage" href="../../configure-pod-container/configure-projected-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Security Context for a Pod or Container" href="../../../user-guide/security-context"></a>

		
	
		
		
<a class="item" data-title="Configure Service Accounts for Pods" href="../../../user-guide/service-accounts"></a>

		
	
		
		
<a class="item" data-title="Pull an Image from a Private Registry" href="../../configure-pod-container/pull-image-private-registry/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Liveness and Readiness Probes" href="../../../user-guide/liveness/index.html"></a>

		
	
		
		
<a class="item" data-title="Assign Pods to Nodes" href="../../configure-pod-container/assign-pods-nodes/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Pod Initialization" href="../../configure-pod-container/configure-pod-initialization/index.html"></a>

		
	
		
		
<a class="item" data-title="Attach Handlers to Container Lifecycle Events" href="../../configure-pod-container/attach-handler-lifecycle-event/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a ConfigMap" href="../../configure-pod-container/configure-pod-configmap/index.html"></a>

		
	
		
		
<a class="item" data-title="Share Process Namespace between Containers in a Pod" href="../../configure-pod-container/share-process-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Translate a Docker Compose File to Kubernetes Resources" href="../../configure-pod-container/translate-compose-kubernetes/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Administer a Cluster">
		<div class="container">
		
		
	
	
		
		
	<div class="item" data-title="Administration with kubeadm">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Upgrading kubeadm HA clusters from 1.9.x to 1.9.y" href="../../administer-cluster/kubeadm/kubeadm-upgrade-ha/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading kubeadm clusters from 1.7 to 1.8" href="../../administer-cluster/kubeadm/kubeadm-upgrade-1-8/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading kubeadm clusters from v1.10 to v1.11" href="../../administer-cluster/kubeadm/kubeadm-upgrade-1-11/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading/downgrading kubeadm clusters between v1.8 to v1.9" href="../../administer-cluster/kubeadm/kubeadm-upgrade-1-9/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Manage Memory, CPU, and API Resources">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Configure Default Memory Requests and Limits for a Namespace" href="../../configure-pod-container/limit-range/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Default CPU Requests and Limits for a Namespace" href="../../administer-cluster/cpu-default-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Minimum and Maximum Memory Constraints for a Namespace" href="../../administer-cluster/memory-constraint-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Minimum and Maximum CPU Constraints for a Namespace" href="../../administer-cluster/cpu-constraint-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Memory and CPU Quotas for a Namespace" href="../../administer-cluster/quota-memory-cpu-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod Quota for a Namespace" href="../../administer-cluster/quota-pod-namespace/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Install a Network Policy Provider">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Use Calico for NetworkPolicy" href="../../administer-cluster/network-policy-provider/calico-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Cilium for NetworkPolicy" href="../../administer-cluster/network-policy-provider/cilium-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Kube-router for NetworkPolicy" href="../../administer-cluster/network-policy-provider/kube-router-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Romana for NetworkPolicy" href="../../administer-cluster/network-policy-provider/romana-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Weave Net for NetworkPolicy" href="../../administer-cluster/network-policy-provider/weave-network-policy/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Access Clusters Using the Kubernetes API" href="../../administer-cluster/access-cluster-api/index.html"></a>

		
	
		
		
<a class="item" data-title="Access Services Running on Clusters" href="../../administer-cluster/access-cluster-services/index.html"></a>

		
	
		
		
<a class="item" data-title="Advertise Extended Resources for a Node" href="../../administer-cluster/extended-resource-node/index.html"></a>

		
	
		
		
<a class="item" data-title="Autoscale the DNS Service in a Cluster" href="../../administer-cluster/dns-horizontal-autoscaling/index.html"></a>

		
	
		
		
<a class="item" data-title="Change the Reclaim Policy of a PersistentVolume" href="../../administer-cluster/change-pv-reclaim-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Change the default StorageClass" href="../../administer-cluster/change-default-storage-class/index.html"></a>

		
	
		
		
<a class="item" data-title="Cluster Management" href="../../../admin/cluster-management/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Multiple Schedulers" href="../../administer-cluster/configure-multiple-schedulers/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Out Of Resource Handling" href="../../administer-cluster/reserve-compute-resources/out-of-resource.md"></a>

		
	
		
		
<a class="item" data-title="Configure Quotas for API Objects" href="../../administer-cluster/quota-api-object/index.html"></a>

		
	
		
		
<a class="item" data-title="Control CPU Management Policies on the Node" href="../../administer-cluster/cpu-management-policies/index.html"></a>

		
	
		
		
<a class="item" data-title="Customizing DNS Service" href="../../administer-cluster/dns-custom-nameservers/index.html"></a>

		
	
		
		
<a class="item" data-title="Debugging DNS Resolution" href="../../administer-cluster/dns-debugging-resolution/index.html"></a>

		
	
		
		
<a class="item" data-title="Declare Network Policy" href="../../configure-pod-container/declare-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Developing Cloud Controller Manager" href="../../administer-cluster/developing-cloud-controller-manager.md"></a>

		
	
		
		
<a class="item" data-title="Encrypting Secret Data at Rest" href="../../administer-cluster/encrypt-data.1"></a>

		
	
		
		
<a class="item" data-title="Guaranteed Scheduling For Critical Add-On Pods" href="../../administer-cluster/guaranteed-scheduling-critical-addon-pods/index.html"></a>

		
	
		
		
<a class="item" data-title="IP Masquerade Agent User Guide" href="../../administer-cluster/ip-masq-agent/index.html"></a>

		
	
		
		
<a class="item" data-title="Kubernetes Cloud Controller Manager" href="../../administer-cluster/running-cloud-controller.md"></a>

		
	
		
		
<a class="item" data-title="Limit Storage Consumption" href="../../administer-cluster/limit-storage-consumption/index.html"></a>

		
	
		
		
<a class="item" data-title="Namespaces Walkthrough" href="../../administer-cluster/namespaces-walkthrough/index.html"></a>

		
	
		
		
<a class="item" data-title="Operating etcd clusters for Kubernetes" href="../../administer-cluster/configure-upgrade-etcd/index.html"></a>

		
	
		
		
<a class="item" data-title="Reconfigure a Node&#39;s Kubelet in a Live Cluster" href="../../administer-cluster/reconfigure-kubelet.1"></a>

		
	
		
		
<a class="item" data-title="Reserve Compute Resources for System Daemons" href="../../administer-cluster/reserve-compute-resources/index.html"></a>

		
	
		
		
<a class="item" data-title="Safely Drain a Node while Respecting Application SLOs" href="../../administer-cluster/safely-drain-node/index.html"></a>

		
	
		
		
<a class="item" data-title="Securing a Cluster" href="../../administer-cluster/securing-a-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Set Kubelet parameters via a config file" href="../../administer-cluster/kubelet-config-file.1"></a>

		
	
		
		
<a class="item" data-title="Set up High-Availability Kubernetes Masters" href="../../administer-cluster/highly-available-master/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up a Highly Availabile etcd Cluster With kubeadm" href="../../administer-cluster/setup-ha-etcd-with-kubeadm/index.html"></a>

		
	
		
		
<a class="item" data-title="Share a Cluster with Namespaces" href="../../../admin/namespaces"></a>

		
	
		
		
<a class="item" data-title="Static Pods" href="../../../concepts/cluster-administration/static-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Storage Object in Use Protection" href="../../administer-cluster/storage-object-in-use-protection/index.html"></a>

		
	
		
		
<a class="item" data-title="Using CoreDNS for Service Discovery" href="../../administer-cluster/coredns/index.html"></a>

		
	
		
		
<a class="item" data-title="Using a KMS provider for data encryption" href="../../administer-cluster/kms-provider/index.html"></a>

		
	
		
		
<a class="item" data-title="Using sysctls in a Kubernetes Cluster" href="../../../concepts/cluster-administration/sysctl-cluster/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Inject Data Into Applications">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Define a Command and Arguments for a Container" href="../../../user-guide/containers/index.html"></a>

		
	
		
		
<a class="item" data-title="Define Environment Variables for a Container" href="../../inject-data-application/define-environment-variable-container/index.html"></a>

		
	
		
		
<a class="item" data-title="Expose Pod Information to Containers Through Environment Variables" href="../../configure-pod-container/environment-variable-expose-pod-information/index.html"></a>

		
	
		
		
<a class="item" data-title="Expose Pod Information to Containers Through Files" href="../../inject-data-application/downward-api-volume-expose-pod-information/index.html"></a>

		
	
		
		
<a class="item" data-title="Distribute Credentials Securely Using Secrets" href="../../inject-data-application/distribute-credentials-secure/index.html"></a>

		
	
		
		
<a class="item" data-title="Inject Information into Pods Using a PodPreset" href="../../inject-data-application/podpreset.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Run Applications">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Run a Stateless Application Using a Deployment" href="../../../user-guide/simple-nginx"></a>

		
	
		
		
<a class="item" data-title="Run a Single-Instance Stateful Application" href="../../../tutorials/stateful-application/run-stateful-application/index.html"></a>

		
	
		
		
<a class="item" data-title="Run a Replicated Stateful Application" href="../../run-application/run-replicated-stateful-application/index.html"></a>

		
	
		
		
<a class="item" data-title="Update API Objects in Place Using kubectl patch" href="../../run-application/update-api-object-kubectl-patch/index.html"></a>

		
	
		
		
<a class="item" data-title="Scale a StatefulSet" href="../../run-application/scale-stateful-set/index.html"></a>

		
	
		
		
<a class="item" data-title="Delete a StatefulSet" href="../../manage-stateful-set/delete-pods/index.html"></a>

		
	
		
		
<a class="item" data-title="Force Delete StatefulSet Pods" href="../../run-application/force-delete-stateful-set-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Perform Rolling Update Using a Replication Controller" href="../../run-application/rolling-update-replication-controller/index.html"></a>

		
	
		
		
<a class="item" data-title="Horizontal Pod Autoscaler" href="../../run-application/horizontal-pod-autoscale/index.html"></a>

		
	
		
		
<a class="item" data-title="Horizontal Pod Autoscaler Walkthrough" href="../../run-application/horizontal-pod-autoscale-walkthrough/index.html"></a>

		
	
		
		
<a class="item" data-title="Specifying a Disruption Budget for your Application" href="../../configure-pod-container/configure-pod-disruption-budget/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Run Jobs">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Running automated tasks with cron jobs" href="../automated-tasks-with-cron-jobs.1"></a>

		
	
		
		
<a class="item" data-title="Parallel Processing using Expansions" href="../parallel-processing-expansion/index.html"></a>

		
	
		
		
<a class="item" data-title="Coarse Parallel Processing Using a Work Queue" href="index.html"></a>

		
	
		
		
<a class="item" data-title="Fine Parallel Processing Using a Work Queue" href="../fine-parallel-processing-work-queue/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Access Applications in a Cluster">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Web UI (Dashboard)" href="../../web-ui-dashboard/index.html"></a>

		
	
		
		
<a class="item" data-title="Accessing Clusters" href="../../../concepts/cluster-administration/access-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Access to Multiple Clusters" href="../../access-application-cluster/authenticate-across-clusters-kubeconfig/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Port Forwarding to Access Applications in a Cluster" href="../../access-application-cluster/port-forward-access-application-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Provide Load-Balanced Access to an Application in a Cluster" href="../../access-application-cluster/load-balance-access-application-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Use a Service to Access an Application in a Cluster" href="../../access-application-cluster/service-access-application-cluster.1"></a>

		
	
		
		
<a class="item" data-title="Connect a Front End to a Back End Using a Service" href="../../access-application-cluster/connecting-frontend-backend/index.html"></a>

		
	
		
		
<a class="item" data-title="Create an External Load Balancer" href="../../../user-guide/load-balancer"></a>

		
	
		
		
<a class="item" data-title="Configure Your Cloud Provider&#39;s Firewalls" href="../../access-application-cluster/configure-cloud-provider-firewall/index.html"></a>

		
	
		
		
<a class="item" data-title="List All Container Images Running in a Cluster" href="../../access-application-cluster/list-all-running-container-images/index.html"></a>

		
	
		
		
<a class="item" data-title="Communicate Between Containers in the Same Pod Using a Shared Volume" href="../../access-application-cluster/communicate-containers-same-pod-shared-volume/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure DNS for a Cluster" href="../../access-application-cluster/configure-dns-cluster/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Monitor, Log, and Debug">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Application Introspection and Debugging" href="../../debug-application-cluster/debug-application-introspection/index.html"></a>

		
	
		
		
<a class="item" data-title="Auditing" href="../../debug-application-cluster/audit/index.html"></a>

		
	
		
		
<a class="item" data-title="Core metrics pipeline" href="../../debug-application-cluster/core-metrics-pipeline/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Init Containers" href="../../debug-application-cluster/debug-init-containers/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Pods and Replication Controllers" href="../../debug-application-cluster/debug-pod-replication-controller/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Services" href="../../../user-guide/debugging-services"></a>

		
	
		
		
<a class="item" data-title="Debug a StatefulSet" href="../../manage-stateful-set/debugging-a-statefulset/index.html"></a>

		
	
		
		
<a class="item" data-title="Debugging Kubernetes nodes with crictl" href="../../debug-application-cluster/crictl/index.html"></a>

		
	
		
		
<a class="item" data-title="Determine the Reason for Pod Failure" href="../../debug-application-cluster/determine-reason-pod-failure/index.html"></a>

		
	
		
		
<a class="item" data-title="Developing and debugging services locally" href="../../debug-application-cluster/local-debugging/index.html"></a>

		
	
		
		
<a class="item" data-title="Events in Stackdriver" href="../../debug-application-cluster/events-stackdriver/index.html"></a>

		
	
		
		
<a class="item" data-title="Get a Shell to a Running Container" href="../../debug-application-cluster/get-shell-running-container/index.html"></a>

		
	
		
		
<a class="item" data-title="Logging Using Elasticsearch and Kibana" href="../../../user-guide/logging/elasticsearch.1"></a>

		
	
		
		
<a class="item" data-title="Logging Using Stackdriver" href="../../../user-guide/logging/stackdriver.1"></a>

		
	
		
		
<a class="item" data-title="Monitor Node Health" href="../../debug-application-cluster/monitor-node-health/index.html"></a>

		
	
		
		
<a class="item" data-title="Tools for Monitoring Compute, Storage, and Network Resources" href="../../debug-application-cluster/resource-usage-monitoring/index.html"></a>

		
	
		
		
<a class="item" data-title="Troubleshoot Applications" href="../../debug-application-cluster/debug-application.1"></a>

		
	
		
		
<a class="item" data-title="Troubleshoot Clusters" href="../../../admin/cluster-troubleshooting.1"></a>

		
	
		
		
<a class="item" data-title="Troubleshooting" href="../../../troubleshooting/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Extend Kubernetes">
		<div class="container">
		
		
	
	
		
		
	<div class="item" data-title="Use Custom Resources">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Extend the Kubernetes API with CustomResourceDefinitions" href="../../access-kubernetes-api/extend-api-custom-resource-definitions/index.html"></a>

		
	
		
		
<a class="item" data-title="Versions of CustomResourceDefinitions" href="../../access-kubernetes-api/custom-resources/custom-resource-definition-versioning/index.html"></a>

		
	
		
		
<a class="item" data-title="Migrate a ThirdPartyResource to CustomResourceDefinition" href="../../access-kubernetes-api/migrate-third-party-resource/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Configure the aggregation layer" href="../../access-kubernetes-api/configure-aggregation-layer/index.html"></a>

		
	
		
		
<a class="item" data-title="Setup an extension API server" href="../../access-kubernetes-api/setup-extension-api-server/index.html"></a>

		
	
		
		
<a class="item" data-title="Use an HTTP Proxy to Access the Kubernetes API" href="../../access-kubernetes-api/http-proxy-access-api.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="TLS">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Certificate Rotation" href="../../tls/certificate-rotation/index.html"></a>

		
	
		
		
<a class="item" data-title="Manage TLS Certificates in a Cluster" href="../../tls/managing-tls-in-a-cluster.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Federation - Run an App on Multiple Clusters">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Cross-cluster Service Discovery using Federated Services" href="../../../concepts/cluster-administration/federation-service-discovery/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up Cluster Federation with Kubefed" href="../../../tutorials/federation/set-up-cluster-federation-kubefed/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up CoreDNS as DNS provider for Cluster Federation" href="../../federation/set-up-coredns-provider-federation/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up placement policies in Federation" href="../../federation/set-up-placement-policies-federation/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Manage Cluster Daemons">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Perform a Rolling Update on a DaemonSet" href="../../manage-daemon/update-daemon-set/index.html"></a>

		
	
		
		
<a class="item" data-title="Performing a Rollback on a DaemonSet" href="../../manage-daemon/rollback-daemon-set/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Install Service Catalog">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Install Service Catalog using Helm" href="../../service-catalog/install-service-catalog-using-helm/index.html"></a>

		
	
		
		
<a class="item" data-title="Install Service Catalog using SC" href="../../service-catalog/install-service-catalog-using-sc/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Federation - Run an App on Multiple Clusters">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Federated Cluster" href="../../administer-federation/cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated ConfigMap" href="../../administer-federation/configmap/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated DaemonSet" href="../../administer-federation/daemonset/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Deployment" href="../../administer-federation/deployment/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Events" href="../../administer-federation/events/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Horizontal Pod Autoscalers (HPA)" href="../../administer-federation/hpa/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Ingress" href="../../administer-federation/ingress/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Jobs" href="../../administer-federation/job/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Namespaces" href="../../administer-federation/namespaces/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated ReplicaSets" href="../../administer-federation/replicaset/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Secrets" href="../../administer-federation/secret/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Extend kubectl with plugins" href="../../extend-kubectl/kubectl-plugins/index.html"></a>

		
	
		
		
<a class="item" data-title="Manage HugePages" href="../../manage-hugepages/scheduling-hugepages/index.html"></a>

		
	
		
		
<a class="item" data-title="Schedule GPUs" href="../../manage-gpus/scheduling-gpus/index.html"></a>

		
	






     </div> 
    <button class="push-menu-close-button" onclick="kub.toggleToc()"></button>
</div> 

			<div id="docsContent">
				
<p><a href="../../../editdocs#docs/tasks/job/coarse-parallel-processing-work-queue.md" id="editPageButton">Edit This Page</a></p>

<h1>Coarse Parallel Processing Using a Work Queue</h1>




<p>In this example, we will run a Kubernetes Job with multiple parallel
worker processes.</p>

<p>In this example, as each pod is created, it picks up one unit of work
from a task queue, completes it, deletes it from the queue, and exits.</p>

<p>Here is an overview of the steps in this example:</p>

<ol>
<li><strong>Start a message queue service.</strong>  In this example, we use RabbitMQ, but you could use another
one.  In practice you would set up a message queue service once and reuse it for many jobs.</li>
<li><strong>Create a queue, and fill it with messages.</strong>  Each message represents one task to be done.  In
this example, a message is just an integer that we will do a lengthy computation on.</li>
<li><strong>Start a Job that works on tasks from the queue</strong>.  The Job starts several pods.  Each pod takes
one task from the message queue, processes it, and repeats until the end of the queue is reached.</li>
</ol>











<ul id="markdown-toc">










<li><a href="index.html#before-you-begin">Before you begin</a></li>












<li><a href="index.html#starting-a-message-queue-service">Starting a message queue service</a></li>




<li><a href="index.html#testing-the-message-queue-service">Testing the message queue service</a></li>




<li><a href="index.html#filling-the-queue-with-tasks">Filling the Queue with tasks</a></li>




<li><a href="index.html#create-an-image">Create an Image</a></li>




<li><a href="index.html#defining-a-job">Defining a Job</a></li>




<li><a href="index.html#running-the-job">Running the Job</a></li>












<li><a href="index.html#alternatives">Alternatives</a></li>




<li><a href="index.html#caveats">Caveats</a></li>



















</ul>



<h2 id="before-you-begin">Before you begin</h2>
<p>Be familiar with the basic,
non-parallel, use of <a href="../../../concepts/jobs/run-to-completion-finite-workloads/index.html">Job</a>.</p>

<p>You need to have a Kubernetes cluster, and the kubectl command-line tool must
be configured to communicate with your cluster. If you do not already have a
cluster, you can create one by using
<a href="../../../getting-started-guides/minikube">Minikube</a>,
or you can use one of these Kubernetes playgrounds:</p>

<ul>
<li><a href="https://www.katacoda.com/courses/kubernetes/playground" target="_blank">Katacoda</a></li>
<li><a href="http://labs.play-with-k8s.com/" target="_blank">Play with Kubernetes</a></li>
</ul>
 

<p>To check the version, enter <code>kubectl version</code>.</p>




<h2 id="starting-a-message-queue-service">Starting a message queue service</h2>

<p>This example uses RabbitMQ, but it should be easy to adapt to another AMQP-type message service.</p>

<p>In practice you could set up a message queue service once in a
cluster and reuse it for many jobs, as well as for long-running services.</p>

<p>Start RabbitMQ as follows:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ kubectl create -f examples/celery-rabbitmq/rabbitmq-service.yaml
service <span style="color:#b44">&#34;rabbitmq-service&#34;</span> created
$ kubectl create -f examples/celery-rabbitmq/rabbitmq-controller.yaml
replicationcontroller <span style="color:#b44">&#34;rabbitmq-controller&#34;</span> created</code></pre></div>
<p>We will only use the rabbitmq part from the <a href="https://github.com/kubernetes/kubernetes/tree/release-1.3/examples/celery-rabbitmq" target="_blank">celery-rabbitmq example</a>.</p>

<h2 id="testing-the-message-queue-service">Testing the message queue service</h2>

<p>Now, we can experiment with accessing the message queue.  We will
create a temporary interactive pod, install some tools on it,
and experiment with queues.</p>

<p>First create a temporary interactive Pod.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#080;font-style:italic"># Create a temporary interactive container
</span><span style="color:#080;font-style:italic"></span>$ kubectl run -i --tty temp --image ubuntu:14.04
Waiting <span style="color:#a2f;font-weight:bold">for</span> pod default/temp-loe07 to be running, status is Pending, pod ready: <span style="color:#a2f">false</span>
... <span style="color:#666">[</span> previous line repeats several <span style="color:#a2f">times</span> .. hit <span style="color:#a2f;font-weight:bold">return</span> when it stops <span style="color:#666">]</span> ...</code></pre></div>
<p>Note that your pod name and command prompt will be different.</p>

<p>Next install the <code>amqp-tools</code> so we can work with message queues.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#080;font-style:italic"># Install some tools
</span><span style="color:#080;font-style:italic"></span>root@temp-loe07:/# apt-get update
.... <span style="color:#666">[</span> lots of output <span style="color:#666">]</span> ....
root@temp-loe07:/# apt-get install -y curl ca-certificates amqp-tools python dnsutils
.... <span style="color:#666">[</span> lots of output <span style="color:#666">]</span> ....</code></pre></div>
<p>Later, we will make a docker image that includes these packages.</p>

<p>Next, we will check that we can discover the rabbitmq service:</p>

<pre><code># Note the rabbitmq-service has a DNS name, provided by Kubernetes:

root@temp-loe07:/# nslookup rabbitmq-service
Server:        10.0.0.10
Address:    10.0.0.10#53

Name:    rabbitmq-service.default.svc.cluster.local
Address: 10.0.147.152

# Your address will vary.
</code></pre>

<p>If Kube-DNS is not setup correctly, the previous step may not work for you.
You can also find the service IP in an env var:</p>

<pre><code># env | grep RABBIT | grep HOST
RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152
# Your address will vary.
</code></pre>

<p>Next we will verify we can create a queue, and publish and consume messages.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#080;font-style:italic"># In the next line, rabbitmq-service is the hostname where the rabbitmq-service
</span><span style="color:#080;font-style:italic"># can be reached.  5672 is the standard port for rabbitmq.
</span><span style="color:#080;font-style:italic"></span>
root@temp-loe07:/# <span style="color:#a2f">export</span> <span style="color:#b8860b">BROKER_URL</span><span style="color:#666">=</span>amqp://guest:guest@rabbitmq-service:5672
<span style="color:#080;font-style:italic"># If you could not resolve &#34;rabbitmq-service&#34; in the previous step,
</span><span style="color:#080;font-style:italic"># then use this command instead:
</span><span style="color:#080;font-style:italic"># root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
</span><span style="color:#080;font-style:italic"></span>
<span style="color:#080;font-style:italic"># Now create a queue:
</span><span style="color:#080;font-style:italic"></span>
root@temp-loe07:/# /usr/bin/amqp-declare-queue --url<span style="color:#666">=</span><span style="color:#b8860b">$BROKER_URL</span> -q foo -d
foo

<span style="color:#080;font-style:italic"># Publish one message to it:
</span><span style="color:#080;font-style:italic"></span>
root@temp-loe07:/# /usr/bin/amqp-publish --url<span style="color:#666">=</span><span style="color:#b8860b">$BROKER_URL</span> -r foo -p -b Hello

<span style="color:#080;font-style:italic"># And get it back.
</span><span style="color:#080;font-style:italic"></span>
root@temp-loe07:/# /usr/bin/amqp-consume --url<span style="color:#666">=</span><span style="color:#b8860b">$BROKER_URL</span> -q foo -c <span style="color:#666">1</span> cat <span style="color:#666">&amp;&amp;</span> <span style="color:#a2f">echo</span>
Hello
root@temp-loe07:/#</code></pre></div>
<p>In the last command, the <code>amqp-consume</code> tool takes one message (<code>-c 1</code>)
from the queue, and passes that message to the standard input of an arbitrary command.  In this case, the program <code>cat</code> is just printing
out what it gets on the standard input, and the echo is just to add a carriage
return so the example is readable.</p>

<h2 id="filling-the-queue-with-tasks">Filling the Queue with tasks</h2>

<p>Now lets fill the queue with some &ldquo;tasks&rdquo;.  In our example, our tasks are just strings to be
printed.</p>

<p>In a practice, the content of the messages might be:</p>

<ul>
<li>names of files to that need to be processed</li>
<li>extra flags to the program</li>
<li>ranges of keys in a database table</li>
<li>configuration parameters to a simulation</li>
<li>frame numbers of a scene to be rendered</li>
</ul>

<p>In practice, if there is large data that is needed in a read-only mode by all pods
of the Job, you will typically put that in a shared file system like NFS and mount
that readonly on all the pods, or the program in the pod will natively read data from
a cluster file system like HDFS.</p>

<p>For our example, we will create the queue and fill it using the amqp command line tools.
In practice, you might write a program to fill the queue using an amqp client library.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ /usr/bin/amqp-declare-queue --url<span style="color:#666">=</span><span style="color:#b8860b">$BROKER_URL</span> -q job1  -d
job1
$ <span style="color:#a2f;font-weight:bold">for</span> f in apple banana cherry date fig grape lemon melon
<span style="color:#a2f;font-weight:bold">do</span>
  /usr/bin/amqp-publish --url<span style="color:#666">=</span><span style="color:#b8860b">$BROKER_URL</span> -r job1 -p -b <span style="color:#b8860b">$f</span>
<span style="color:#a2f;font-weight:bold">done</span></code></pre></div>
<p>So, we filled the queue with 8 messages.</p>

<h2 id="create-an-image">Create an Image</h2>

<p>Now we are ready to create an image that we will run as a job.</p>

<p>We will use the <code>amqp-consume</code> utility to read the message
from the queue and run our actual program.  Here is a very simple
example program:</p>

<table class="includecode" id="coarse-parallel-processing-work-queue-worker-py">
    <thead>
        <tr>
            <th>
                <a href="https://github.com/kubernetes/website/blob/master/content/en/docs/tasks/job/coarse-parallel-processing-work-queue/worker.py" download="coarse-parallel-processing-work-queue/worker.py">
                    <code>coarse-parallel-processing-work-queue/worker.py docs/tasks/job/coarse-parallel-processing-work-queue</code>
                </a>
                <img src="../../../../images/copycode.svg" style="max-height:24px" onclick="copyCode('coarse-parallel-processing-work-queue-worker-py')" title="Copy coarse-parallel-processing-work-queue/worker.py to clipboard">
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-style:italic">#!/usr/bin/env python</span>

<span style="color:#080;font-style:italic"># Just prints standard out and sleeps for 10 seconds.</span>
<span style="color:#a2f;font-weight:bold">import</span> <span style="color:#00f;font-weight:bold">sys</span>
<span style="color:#a2f;font-weight:bold">import</span> <span style="color:#00f;font-weight:bold">time</span>
<span style="color:#a2f;font-weight:bold">print</span>(<span style="color:#b44">&#34;Processing &#34;</span> <span style="color:#666">+</span> sys<span style="color:#666">.</span>stdin<span style="color:#666">.</span>lines())
time<span style="color:#666">.</span>sleep(<span style="color:#666">10</span>)
</code></pre></div>  </td>
        </tr>
    </tbody>
</table>

<p>Now, build an image.  If you are working in the source
tree, then change directory to <code>examples/job/work-queue-1</code>.
Otherwise, make a temporary directory, change to it,
download the <a href="Dockerfile?raw=true" target="_blank">Dockerfile</a>,
and <a href="worker.py?raw=true" target="_blank">worker.py</a>.  In either case,
build the image with this command:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ docker build -t job-wq-1 .</code></pre></div>
<p>For the <a href="https://hub.docker.com/" target="_blank">Docker Hub</a>, tag your app image with
your username and push to the Hub with the below commands. Replace
<code>&lt;username&gt;</code> with your Hub username.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">docker tag job-wq-1 &lt;username&gt;/job-wq-1
docker push &lt;username&gt;/job-wq-1</code></pre></div>
<p>If you are using <a href="https://cloud.google.com/tools/container-registry/" target="_blank">Google Container
Registry</a>, tag
your app image with your project ID, and push to GCR. Replace
<code>&lt;project&gt;</code> with your project ID.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">docker tag job-wq-1 gcr.io/&lt;project&gt;/job-wq-1
gcloud docker -- push gcr.io/&lt;project&gt;/job-wq-1</code></pre></div>
<h2 id="defining-a-job">Defining a Job</h2>

<p>Here is a job definition.  You&rsquo;ll need to make a copy of the Job and edit the
image to match the name you used, and call it <code>./job.yaml</code>.</p>

<table class="includecode" id="coarse-parallel-processing-work-queue-job-yaml">
    <thead>
        <tr>
            <th>
                <a href="https://github.com/kubernetes/website/blob/master/content/en/docs/tasks/job/coarse-parallel-processing-work-queue/job.yaml" download="coarse-parallel-processing-work-queue/job.yaml">
                    <code>coarse-parallel-processing-work-queue/job.yaml docs/tasks/job/coarse-parallel-processing-work-queue</code>
                </a>
                <img src="../../../../images/copycode.svg" style="max-height:24px" onclick="copyCode('coarse-parallel-processing-work-queue-job-yaml')" title="Copy coarse-parallel-processing-work-queue/job.yaml to clipboard">
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">apiVersion:<span style="color:#bbb"> </span>batch/v1<span style="color:#bbb">
</span><span style="color:#bbb"></span>kind:<span style="color:#bbb"> </span>Job<span style="color:#bbb">
</span><span style="color:#bbb"></span>metadata:<span style="color:#bbb">
</span><span style="color:#bbb">  </span>name:<span style="color:#bbb"> </span>job-wq-<span style="color:#666">1</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>spec:<span style="color:#bbb">
</span><span style="color:#bbb">  </span>completions:<span style="color:#bbb"> </span><span style="color:#666">8</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span>parallelism:<span style="color:#bbb"> </span><span style="color:#666">2</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span>template:<span style="color:#bbb">
</span><span style="color:#bbb">    </span>metadata:<span style="color:#bbb">
</span><span style="color:#bbb">      </span>name:<span style="color:#bbb"> </span>job-wq-<span style="color:#666">1</span><span style="color:#bbb">
</span><span style="color:#bbb">    </span>spec:<span style="color:#bbb">
</span><span style="color:#bbb">      </span>containers:<span style="color:#bbb">
</span><span style="color:#bbb">      </span>-<span style="color:#bbb"> </span>name:<span style="color:#bbb"> </span>c<span style="color:#bbb">
</span><span style="color:#bbb">        </span>image:<span style="color:#bbb"> </span>gcr.io/&lt;project&gt;/job-wq-<span style="color:#666">1</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span>env:<span style="color:#bbb">
</span><span style="color:#bbb">        </span>-<span style="color:#bbb"> </span>name:<span style="color:#bbb"> </span>BROKER_URL<span style="color:#bbb">
</span><span style="color:#bbb">          </span>value:<span style="color:#bbb"> </span>amqp://guest:guest@rabbitmq-service:<span style="color:#666">5672</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span>-<span style="color:#bbb"> </span>name:<span style="color:#bbb"> </span>QUEUE<span style="color:#bbb">
</span><span style="color:#bbb">          </span>value:<span style="color:#bbb"> </span>job1<span style="color:#bbb">
</span><span style="color:#bbb">      </span>restartPolicy:<span style="color:#bbb"> </span>OnFailure<span style="color:#bbb">
</span><span style="color:#bbb"></span></code></pre></div>  </td>
        </tr>
    </tbody>
</table>

<p>In this example, each pod works on one item from the queue and then exits.
So, the completion count of the Job corresponds to the number of work items
done.  So we set, <code>.spec.completions: 8</code> for the example, since we put 8 items in the queue.</p>

<h2 id="running-the-job">Running the Job</h2>

<p>So, now run the Job:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">kubectl create -f ./job.yaml</code></pre></div>
<p>Now wait a bit, then check on the job.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ kubectl describe jobs/job-wq-1
Name:             job-wq-1
Namespace:        default
Selector:         controller-uid<span style="color:#666">=</span>41d75705-92df-11e7-b85e-fa163ee3c11f
Labels:           controller-uid<span style="color:#666">=</span>41d75705-92df-11e7-b85e-fa163ee3c11f
                  job-name<span style="color:#666">=</span>job-wq-1
Annotations:      &lt;none&gt;
Parallelism:      <span style="color:#666">2</span>
Completions:      <span style="color:#666">8</span>
Start Time:       Wed, <span style="color:#666">06</span> Sep <span style="color:#666">2017</span> <span style="color:#666">16</span>:42:02 +0800
Pods Statuses:    <span style="color:#666">0</span> Running / <span style="color:#666">8</span> Succeeded / <span style="color:#666">0</span> Failed
Pod Template:
  Labels:       controller-uid<span style="color:#666">=</span>41d75705-92df-11e7-b85e-fa163ee3c11f
                job-name<span style="color:#666">=</span>job-wq-1
  Containers:
   c:
    Image:      gcr.io/causal-jigsaw-637/job-wq-1
    Port:
    Environment:
      BROKER_URL:       amqp://guest:guest@rabbitmq-service:5672
      QUEUE:            job1
    Mounts:             &lt;none&gt;
  Volumes:              &lt;none&gt;
Events:
  FirstSeen  LastSeen   Count    From    SubobjectPath    Type      Reason              Message
  ─────────  ────────   ─────    ────    ─────────────    ──────    ──────              ───────
  27s        27s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-hcobb
  27s        27s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-weytj
  27s        27s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-qaam5
  27s        27s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-b67sr
  26s        26s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-xe5hj
  15s        15s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-w2zqe
  14s        14s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-d6ppa
  14s        14s        <span style="color:#666">1</span>        <span style="color:#666">{</span>job <span style="color:#666">}</span>                   Normal    SuccessfulCreate    Created pod: job-wq-1-p17e0</code></pre></div>
<p>All our pods succeeded.  Yay.</p>




<h2 id="alternatives">Alternatives</h2>

<p>This approach has the advantage that you
do not need to modify your &ldquo;worker&rdquo; program to be aware that there is a work queue.</p>

<p>It does require that you run a message queue service.
If running a queue service is inconvenient, you may
want to consider one of the other <a href="../../../concepts/jobs/run-to-completion-finite-workloads/index.html#job-patterns">job patterns</a>.</p>

<p>This approach creates a pod for every work item.  If your work items only take a few seconds,
though, creating a Pod for every work item may add a lot of overhead.  Consider another
<a href="../fine-parallel-processing-work-queue/index.html">example</a>, that executes multiple work items per Pod.</p>

<p>In this example, we used use the <code>amqp-consume</code> utility to read the message
from the queue and run our actual program.  This has the advantage that you
do not need to modify your program to be aware of the queue.
A <a href="../fine-parallel-processing-work-queue/index.html">different example</a>, shows how to
communicate with the work queue using a client library.</p>

<h2 id="caveats">Caveats</h2>

<p>If the number of completions is set to less than the number of items in the queue, then
not all items will be processed.</p>

<p>If the number of completions is set to more than the number of items in the queue,
then the Job will not appear to be completed, even though all items in the queue
have been processed.  It will start additional pods which will block waiting
for a message.</p>

<p>There is an unlikely race with this pattern.  If the container is killed in between the time
that the message is acknowledged by the amqp-consume command and the time that the container
exits with success, or if the node crashes before the kubelet is able to post the success of the pod
back to the api-server, then the Job will not appear to be complete, even though all items
in the queue have been processed.</p>






















				<div class="issue-button-container">
					<p><a href="index.html"><img src="https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/tasks/job/coarse-parallel-processing-work-queue.md?pixel" alt="Analytics" /></a></p>
					
					
					<script type="text/javascript">
					PDRTJS_settings_8345992 = {
					"id" : "8345992",
					"unique_id" : "\/docs\/tasks\/job\/coarse-parallel-processing-work-queue\/",
					"title" : "Coarse Parallel Processing Using a Work Queue",
					"permalink" : "https:\/\/kubernetes.io\/docs\/tasks\/job\/coarse-parallel-processing-work-queue\/"
					};
					(function(d,c,j){if(!document.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src=('https:'==document.location.protocol)?'https://polldaddy.com/js/rating/rating.js':'http://i0.poll.fm/js/rating/rating.js';s=document.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);}}(document,'script','pd-rating-js'));
					</script>
					<a href="index.html" onclick="window.open('https://github.com/kubernetes/website/issues/new?title=Issue%20with%20' +
					'k8s.io'+window.location.pathname)" class="button issue">Create an Issue</a>
					
					
					
					<a href="../../../editdocs#docs/tasks/job/coarse-parallel-processing-work-queue.md" class="button issue">Edit this Page</a>
					
				</div>
			</div>
		</section>
		<footer>
    <main class="light-text">
        <nav>
            
            
            
            <a href="../../../home.1">Documentation</a>
            
            <a href="../../../../blog/index.html">Blog</a>
            
            <a href="../../../../partners/index.html">Partners</a>
            
            <a href="../../../../community/index.html">Community</a>
            
            <a href="../../../../case-studies/index.html">Case Studies</a>
            
        </nav>
        <div class="social">
            <div>
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
            </div>
            <div>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
            <div>
                <a href="../../../getting-started-guides/index.html" class="button">Get Kubernetes</a>
                <a href="https://git.k8s.io/community/contributors/guide" class="button">Contribute</a>
            </div>
        </div>
        <div id="miceType" class="center">
            &copy; 2018 The Kubernetes Authors | Documentation Distributed under <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a>
        </div>
        <div id="miceType" class="center">
            Copyright &copy; 2018 The Linux Foundation&reg;. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">Trademark Usage page</a>
        </div>
    </main>
</footer>

		<button class="flyout-button" onclick="kub.toggleToc()"></button>

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-36037335-10', 'auto');
ga('send', 'pageview');


(function () {
    window.addEventListener('DOMContentLoaded', init)

        
        function init() {
            window.removeEventListener('DOMContentLoaded', init)
                hideNav()
        }

    function hideNav(toc){
        if (!toc) toc = document.querySelector('#docsToc')
        if (!toc) return
            var container = toc.querySelector('.container')

                
                if (container) {
                    if (container.childElementCount === 0 || toc.querySelectorAll('a.item').length === 1) {
                        toc.style.display = 'none'
                            document.getElementById('docsContent').style.width = '100%'
                    }
                } else {
                    requestAnimationFrame(function () {
                        hideNav(toc)
                    })
                }
    }
})();
</script>



	</body>
</html>